HDFS系列(5)|进行API操作前的准备

本篇博文给大家介绍的是进行JavaAPI操作前的准备工作

个人可根据自己电脑的操作系统下载对应的hadoop.jar包到非中文路径

所用jar包的连接如下&＃xff1a;
链接&＃xff1a;https://pan.baidu.com/s/1krbLrnejfXSsLppauxrCTg
提取码&＃xff1a;jbsr

一. 电脑安装配置Hadoop

1. 将下载的jar包放到非中文路径

博主路径为&＃xff1a;D:\hadoop\hadoop-2.7.2

2. 配置HADOOP_HOME环境变量

3. 配置Path环境变量

4. 测试是否成功

输入&＃xff1a;winutils

如果出现上述界面&＃xff0c;即代表成功&＃xff0c;否则为失败

二. maven

1. 此步可省略&＃xff0c;一般用自带的即可

2. 创建一个Maven工程

• 1. 创建过程

• 2. 添加依赖
  在pom.xml中添加如下内容:

<dependencies><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>RELEASE</version></dependency><dependency><groupId>org.apache.logging.log4j</groupId><artifactId>log4j-core</artifactId><version>2.8.2</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-common</artifactId><version>2.7.2</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>2.7.2</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-hdfs</artifactId><version>2.7.2</version></dependency><dependency><groupId>jdk.tools</groupId><artifactId>jdk.tools</artifactId><version>1.8</version><scope>system</scope><systemPath>D:/java/jdk-1.8.0/lib/tools.jar</systemPath></dependency></dependencies>


• 3. 添加完毕之后,右上角检查没有报错说明我们的maven项目的jar包配置成功!

• 4. 如果Eclipse/Idea打印不出日志&＃xff0c;在控制台上只显示&＃xff1a;

1.log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
2.log4j:WARN Please initialize the log4j system properly.
3.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

需要在项目的src/main/resources目录下&＃xff0c;新建一个文件&＃xff0c;命名为“log4j.properties”&＃xff0c;在文件中填入

log4j.rootLogger&＃61;INFO, stdout
log4j.appender.stdout&＃61;org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout&＃61;org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern&＃61;%d %p [%c] - %m%n
log4j.appender.logfile&＃61;org.apache.log4j.FileAppender
log4j.appender.logfile.File&＃61;target/spring.log
log4j.appender.logfile.layout&＃61;org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern&＃61;%d %p [%c] - %m%n

三. 创建包和类

1. 创建

2. 输入测试代码

package com.buwenbuhuo.hdfs;import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.junit.Test;import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;/*** &＃64;author buwenbuhuo* &＃64;create 2020-04-22 16:45* com.buwenbuhuo.hdfs - the name of the target package where the new class or interface will be created.* hdfs0422 - the name of the current project.*/
public class HDFSClient {&＃64;Testpublic void testMkdirs() throws IOException, InterruptedException, URISyntaxException {// 1 获取文件系统Configuration configuration &＃61; new Configuration();// 配置在集群上运行FileSystem fs &＃61; FileSystem.get(URI.create("hdfs://hadoop001:9000"), configuration, "bigdata");// 2 创建目录fs.mkdirs(new Path("/0422/buwenbuhuo/test"));// 3 关闭资源fs.close();}}

在集群上查看是否成功

本篇博文到这里就结束了,小伙伴们有什么疑惑或好的建议可以积极在评论区留言&＃xff0c;博主会在后续继续推出HDFS系列的其他内容&＃xff0c;希望大家持续关注博主~~